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ABSTRACT 



A study examined the varying degrees to which learners of 
Spanish at the intermediate and advanced levels use paragraph-like structure 
in their answers during a simulated oral proficiency exam. Subjects were 24 
students preparing for the Spanish Texas Oral Proficiency Test. Data from 
simulated oral proficiency interviews were compared to a model template 
reflecting the organization of most advanced- level learners for this task. 
Analysis reveals that the advanced learners provided more propositions and 
supporting statements, closely following the proposed model. Intermediate 
students provided fewer propositions and supporting statements, and either do 
not complete the paradigm or begin to diverge from their original intent. The 
less proficient students usually worked at the level of the single 
proposition with single supporting statement, and used other strategies to 
compensate for their difficulties. It is proposed that the template approach 
be used in the assessment of foreign language oral proficiency in the 
classroom, since it can reveal more of the global expression of ideas, as 
contrasted with the evaluation of discrete items such as verb conjugation and 
agreement errors. (MSE) 
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A Discourse Approach to the Assessment of Foreign Language Oral 
Proficiency 

DALE APRIL KOIKE, The University of Texas at Austin 

FANNY HINOJOSA, Universidad Autonoma de Nuevo Leon, Monterrey 

This study examines the varying degrees to which learners of Spanish at 
the Intermediate and Advanced levels use paragraph-like structure in 
their answers during a simulated oral proficiency examination. W e 
compare the structure of the data to a model template that reflects the 
organization of most Advanced-level learners for this task. An analysis 
reveals that the Advanced learners in the study provide more proposi- 
tions and supporting statements to complete the paradigm, closely fol- 
lowing the proposed model. The Intermediate testees provide fewer 
propositions and supporting statements and either do not complete the 
paradigm or begin to diverge from their original intent. The less profi- 
cient speakers usually work at the level of the single proposition with 
single supporting statement, and they use other strategies to attempt to 
compensate for their deficiencies. We propose that the template ap- 
proach be used in the assessment of foreign language oral proficiency in 
the classroom, since it can reveal more of the global expression of ideas, 
as opposed to the evaluation of discrete items such as verb conjugation 
and agreement errors. 

INTRODUCTION 

The focus on communicative activities in the foreign language class- 
room that allow learners to use the target language in meaningful, interac- 
tive, and engaging ways has made a profound impact on the way classes are 
taught, on expectations on the part of teachers and learners, on the content 
and format of textbooks, and on the organization of class time and environ- 
ment. It has inspired a change in the way we assess language proficiency in a 
global sense, seen for example in the American Council on the Teaching of 
Foreign Languages (ACTFL) Oral Proficiency Interview. What is still lacking 
is a reliable, valid, and time- and cost-efficient form of testing that can be used 
in the foreign language classroom and that truly reflects the communicative 
goals that are set out for the learners. 

A proficiency test is not particularly useful for course assessment because 
it is not bound to the curriculum and does not "cover a pre-established body 
of content assigned beforehand" (Higgs, 1987, p. 284). Its purpose is to measure 
a testee's performance against what a native speaker might be expected to say 
in the same situation. Higgs states that the proficiency test is not an appropri- 
ate tool for rank ordering a group of students, since the ratings represent 
ranges of performance and cannot be used productively to make fine distinc- 
tions seen in grading systems in this country. The interviewer notes patterns 
of strengths and weaknesses instead of individual right or wrong answers. 
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This type of test format is not in- 
tended to examine discrete points, 
but rather integrative efforts to ex- 
press a meaningful whole message. 

Teachers can also use an 
achievement test, which covers an 
explicit, finite amount of content 
and is intended to rank the students 
who take the exam. The typical 
achievement test tends to target dis- 
crete points of grammar and vocabu- 
lary, but the problem, as Higgs states, 
is that "the emphasis in achieve- 
ment tests is almost always exclu- 
sively on form rather than function, 
on being correct rather than having 
something interesting, important, or 
true to say" (p. 283). Certainly it is 
the goal of the classroom test to 
cover what is taught. In a skill- 
focused class such as an advanced 
conversation course, what is taught 
is a number of ways in which learn- 
ers can improve their fluency in the 
speaking skill, whether die im- 
provement be in terms of vocabu- 
lary, grammar, organization, or sim- 
ply increased opportunity to speak in 
the target language. The ultimate 
goal is an increase in overall profi- 
ciency, which is difficult to pinpoint 
in terms of an achievement test. 

Another option is the 
"prochievement test," a hybrid test 
that elicits discrete points of gram- 
mar and vocabulary within the con- 
text of some kind of oral interview 
format. It is our belief that such a 
test, as currently defined, is not the 
answer for the conversation course. 
The prochievement test, at least as 
described by Hendrickson (1992), is 
still achievement oriented in that it 
is not aimed at assessing a body of 
discourse in a holistic way, since the 
focus is still on whether or not a tes- 



tee uses a given set of items in re- 
sponding to a stimulus. 

We will describe in this study a 
method of assessment of ongoing 
development for the upper-level 
Spanish conversation course, a 
course in which the focus is the or- 
ganization of learner's discourse. We 
will show that, by looking at the 
structure of the discourse, one can 
see the learner's production in a 
more constructive, top-down, per- 
formance-based way, and the more 
discrete points of grammar are con- 
textualized in meaningful commu- 
nication. 

To this end, we present our re- 
search questions, data from various 
testees at the Intermediate and Ad- 
vanced levels, and an analysis of 
these data. We conclude with some 
suggestions for oral proficiency as- 
sessment in the advanced conversa- 
tion class. 

RESEARCH QUESTIONS 

The questions addressed in this 
study are the following: 

1. What differences are seen in dis- 
course structure produced by 
various learners at Intermediate 
and Advanced levels of Spanish 
proficiency on a simulated oral 
proficiency exam? 

2. What is the applicability of a 
structure-focused oral proficiency 
exam for classroom purposes? 

METHODOLOGY OF DATA 
COLLECTION 

To answer these questions, data 
were collected from an oral profi- 
ciency test used in an advanced 
Spanish conversation course. The 
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following is a description of the test 
instrument, the testees, and the data 
collection and analysis procedure. 

The Test Instrument 

The data for this study are 
drawn from a practice simulated oral 
proficiency exam administered in a 
language laboratory at the end of a 
semester-long advanced Spanish 
conversation course aimed at pre- 
paring potential Spanish language 
teachers of the elementary and high 
school levels to take the Spanish 
Texas Oral Proficiency Test (TOPT). 
This test is required of all prospec- 
tive elementary and secondary Span- 
ish and bilingual education teachers 
in the state of Texas, and they must 
be rated according to their perform- 
ance on the test at the Advanced 
level to pass and receive certification 
to teach. The practice tests, given in 
the Spanish-American variety of 
Spanish, were created to simulate as 
closely as possible the content, func- 
tions, and format of the TOPT 
(Stansfield, 1993; Stansfield & Ken- 
yon, 1992). A difference between the 
practice and live TOPT that should 
be noted is that, on the practice tests, 
testees are usually given 2 minutes 
to answer each question, approxi- 
mately 30 seconds more than on the 
typical Advanced-level question. 

This study focuses on one 
stimulus that presumably requires 
more linguistic skill than others and 
is considered to be of the Advanced 
level: specifically, that of comparing 
advantages and disadvantages. Suc- 
cessful performance on this task is 
considered to be indicative of the 
Advanced level of proficiency ac- 
cording to the ACTFL Guidelines. 







The actual stimulus used is given 
below: 

Stimulus: 

While talking with some of your 
friends in Mexico, one of them 
says that people should use bicy- 
cles as their major means of 
transportation, while another 
friend says that motorcycles 
would solve many problems. 
Then the others present turn to 
you. Present your position logi- 
cally by comparing advantages 
and disadvantages of the bicy- 
cle/motorcycle issue. (20-second 
pause) (Prompt: fY tu? iQue 
crees? 'And what do you think?') 

As in the original version of 
the TOPT, testees heard the test 
stimuli in English over a headset in 
individual language laboratory 
booths and recorded their answers in 
Spanish onto individual audio cas- 
sette tapes. The tapes were graded by 
the instructor for features such as 
fluency, grammar, vocabulary, and 
appropriateness of language, but 
mainly for organization, as part of 
the final grade for the course. 

Testees 

Since the 24 testees were stu- 
dents enrolled in a course designed 
to prepare them to take the TOPT, 
one can assume that, at the point in 
the course at which data were col- 
lected, these testees were more 
skilled at taking this kind of test 
than the average testee. All the tes- 
tees in this study were estimated to 
be at least at the Intermediate level 
of oral proficiency, according to the 
ACTFL Guidelines (1997). Each 
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learner was screened for an ap- 
proximate proficiency level and 
other background factors (for exam- 
ple, true intent to take the TOPT) by 
the instructor before being allowed 
to register for the course. The group 
included six Hispanic Americans, 
two of whom were estimated to be at 
the Advanced level. 

The course these learners were 
enrolled in focused on helping them 
prepare for the TOPT by working on 
rhetorical structure and organiza- 
tional skills, and, most of all, by prac- 
ticing the language functions and 
the format of the test. The learners 
typically were very motivated to 
work hard in the class and took the 
testing procedures seriously since 
they needed to pass the TOPT in or- 
der to qualify for a teaching job. 

Data Collection and Analysis 

In order to look at the organiza- 
tion of the discourse produced by the 
testees, we compared their responses 
to two models of how a logical, top- 
down organizational structure of 
propositions and supporting state- 
ments might appear in the task of 
comparing advantages and disad- 
vantages. These models were com- 
piled from our overall impression of 
what many native speakers of Span- 
ish do to complete this task, and 
what most Advanced and Advanced 
High speakers appear to be attempt- 
ing to do when they answer under 
these test conditions. 

Table 1 illustrates both of the 
two models. Table 1A shows a listing 
strategy, by which the testee lists 
several advantages and disadvan- 
tages of a given referent before doing 
the same for the second referent. 
The supportive statements in paren- 
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theses indicate possible examples 
and other elaboration for those 
points. Table IB provides a contrast- 
ing strategy, by which the testee 
gives the advantages of a referent 
and follows with a contrastive 
statement of the disadvantages of 
the other referent. Then the testee 
gives the advantages of the second 
referent and follows with another 
contrastive statement of the disad- 
vantages of the first referent. In both 
models the paradigm of giving both 
advantages and disadvantages is 
completed, followed by a statement 
of the testee's opinion as to which 
referent is perceived as better. One 
can assume that the more the testees 
complete the paradigm in a logical 
progression, give points of compari- 
son or contrast, and support those 
points with examples and other 
elaboration, the more complete and 
well presented the discourse will be. 
One can also assume that the ability 
to do this is linked to proficiency in 
the target language. 

The transcription of each tes- 
tee's response was analyzed with re- 
spect to the presentation of a topic 
sentence, advantages and disadvan- 
tages with supporting statements, 
and other elaboration, as well as any 
concluding remarks. Each of the in- 
vestigators completed her own 
analysis of the data, which was then 
compared to the other's. Discrepan- 
cies were discussed after all data 
were analyzed. 

RESULTS 

We begin our presentation of 
the results with sample replies from 
three testees who illustrate different 
levels of responses, followed by a 
brief discussion of what we found in 
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Table 1 

Models of Propositional Organization for Comparing 
Advantages and Disadvantages 

Model 1A - Listing Strategy 

Topic sentence: Being a woman has advantages and disadvantages over being 
a man 

Advantage 1 of X: Women are more intelligent 
Advantage 2 of X: Women are more sensitive 

Disadvantage 1 of X: Women are not as physically strong 

(Support: You don't see as many women in professions requiring 
physical strength) 

Disadvantage 2 of X: 

Advantage 1 of Y: Men are strong and tend to be more assertive. 
Advantage 2 of Y: Men are more independent. 

(Support: They get to do more because they don't feel so dependent on 
others to act) 

Disadvantage 1 of Y: Men are not as intelligent. 

Disadvantage 2 of Y: 

Conclusion: It is more advantageous to be a woman. 

Model IB - Contrastive Strategy 

Topic sentence: Being a woman has advantages and disadvantages over being 
a man 

Advantage 1 of X: Women are more intelligent 
Advantage 2 of X: Women are more sensitive 

Disadvantage 1 of Y: Men are insensitive 
Disadvantage 2 of Y: Men think they know everything 

Advantage 1 of Y: Men are physically stronger 
(Support: Men can lift heavy things) 

Advantage 2 of Y: Men are more independent 

Disadvantage 1 of X: Women are not as physically strong 
Disadvantage 2 of X: Women take too long to get ready in the morning 

Conclusion: Still, it's more advantageous to be a woman than a man. 
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them as compared to our models. 
We then discuss categories seen in 
the corpus as a whole and conclude 
with some implications for class- 
room use. 

According to the structural 
quality of responses to the rather dif- 
ficult test item, the testees were di- 



vided into three general groups: 
very complete answers, fairly com- 
plete, and skeletal, with discrepan- 
cies noted within each of the groups. 
An example of a very complete an- 
swer is that of Grace, as shown in 
Example 1: 



Example 1. Grace's Reply: Comparing Advantages and Disadvantages [2]: 

1 Grace: Pienso que, usando una bicicleta o una motocicleta los dos tienen, aventajas y desventajas. 

2 Una mayor, aventaja para la bicicleta pienso es que, la bicicleta es muy bien para el ambiente. 

3 El bicicleta no, no sale, no: saca humo ni nada y es, eso es muy bien. Y pienso que tambien la bici- 

4 cleta es bien para, para hacer ejer*cidos0. ( Uh, una: aventaja para la motocicleta, sera, seria 

5 que, ((sniff)) es muy pequena, y no, nece — no: (0.5) tiene, no es tan grande como un carro. Pero una 

6 desventaja de: la motocicleta es que no es bien para el ambiente. Y las motocicletas hacen mucho 

7 ruido. Pienso yo que, la mejor idea seria usar una bicicleta porque, la bicicleta >pienso que 

8 tiene< mis, aventajas, que la, que la motocicleta.>La bicicleta te puede dar< ejercicios, (0.5) te 

9 puede llevar a muchos lugares una, una desventaja de la bicicleta a lo mejor seria que, el tiempo. 

10 Que en una moticicleta llegarias en un lugar como en diez minutos, pero por bicicleta a lo mejor, 

11 (0.5) neces — vas a necesitar mis tiempo, para llegar adonde necesitas llegar y, a lo mejor si hace 

12 mucho calor afuera, vas a sudar y, a lo mejor eso no es, no es una manera de, de, transportacidn 

13 para Ud. So: pienso que a lo mejor la motocicleta seri, una: (0.3) teng — tenga una aventaja para 

14 las personas que nece si tan llegar a lugares, con aprisa o no, o no necesita-o no tienen que, no 

15 tienen el tiempo para, para gastar en una bicicleta. 

'I think that, using a bicycle or a motorcycle the two have, advantages and dis- 
advantages. A greater, advantage for the bicycle I think is that, the bicycle is 
very fine for the environment. The bicycle doesn't, doesn't leave, doesn't let 
out smoke or anything and it is, that is very fine. I think that also the bicycle 
is fine for, for doing exercise. Uh, an, advantage for the motorcycle, will be, 
would be, is it's very small, and doesn't nee-doesn't have, isn't as big as a car. 
But a disadvantage of, the motorcycle is that it isn't fine for the environment. 
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And motorcycles make a lot of noise. I think that, the best idea would be to 
use a bicycle because, the bicycle I think that it has more, advantages, than the, 
than the motorcycle. The bicycle can give you exercises, can take you to many 
places a, a disadvantage of the bicycle probably would be that, the time. That 
on a motorcycle you would arrive to a place like in ten minutes, but by bicycle 
probably, you nee — you are going to need more time, to arrive where you 
need to arrive and, probably if it's very hot outside, you're going to sweat and, 
probably that isn't, isn't a means of of transportation for you. So, I think that 
probably the motorcycle will be, a, may — may have an advantage for the peo- 
ple who need to get to places, hurriedly or not, or don't nee — or don't have to, 
don't have the time to, to waste on a bicycle.' [3] 



Table 2 shows a structural 
analysis of Grace's response. Num- 
bers represent the order in which the 
main propositions appeared, while 
the words in parentheses represent 
elaborations or supporting state- 
ments. Grace gave a general, organ- 
izing statement at the beginning of 
her reply, stated several advantages 
and disadvantages for both bicycles 



and motorcycles, contrasting posi- 
tive and negative points, and pro- 
vided several supporting statements. 
She was also able to make a summa- 
rizing statement in her final sen- 
tence (Lines 13-15). Grace's discourse 
approximates the model given in 
Table IB (Contrastive Strategy) for 
comparing advantages and disad- 
vantages. 



Table 2 

Grace's Propositional Organization for Comparing 
Advantages and Disadvantages 

Topic sentence: Los dos tienen aventajas y desventajas 'Both have advantages 

and disadvantages' 

Advantage 1 of bicycles: No saca humo 'It doesn't let off smoke' 

Advantage 2 of bicycles: Es bien para hacer ejercicios 'It's fine for doing ex- 
ercise' 

Advantage 1 of motorcycles: Es muy pequeha (No es tan grande como un 
carro) 'It's very small (It isn't as big as a car)' 

Disadvantage 1 of motorcycles: No es bien para el ambiente 'It is not fine 
for the environment' 

Disadvantage 2 of motorcycles: Hacen mucho ruido. 'They make a lot of 
noise' 



(Continued) 
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Table 2 (Continued) 



Topic sentence 2: La bicicleta tiene mas aventajas que la motocicleta 'The bicy- 
cle has more advantages than the motorcycle' 

Advantage 3 of bicycles: Te puede llevar a muchos lugares 'It can take you 
to many places' 

Disadvantage 1 of bicycles: El tiempo. (Vas a necesitar mas tiempo) 'The 
time (You are going to need more time)' 

Disadvantage 2 of bicycles: Vas a sudar. 'You're going to sweat' 

Conclusion: Motocicleta tenga una aventaja para las personas que necesitan 
llegar a lugares con aprisa 'Motorcycle has an advantage for the people 
who need to get to places in a hurry' 



Let's look now at Example 2, the tive of the "fairly complete" group, 

response by a student named Mary but one not as tightly structured as 

who provided a good reply, reflec- that of Grace's: 



Example 2. Mary's Reply for Comparing Advantages and Disadvantages: 

1 Mary: Ento:nces, uh, >mis amigosc, creo que, uh, la bicicleta es, es la solucion, uh, mejor, para el 

2 problema de viajar. Porque, uh, creo que la bicicleta? Uh, no da, uh mas palus—, polucion, a la 

3 dudad, en Mexico, y, um el, la, motodcleta, si, um da, la, polucion, urn, el. Aunque la bi — la mo** 

4 tocicleta es, es mas rapido, um, para viajar, la (bicicleta (no requiere, uh, no requiere tiempo o 

5 dinero, o demasiado tiempo o dinero para arreglar y para segurar que, uh, esta funcionada, uh, 

6 todo el tiempo. Um (0.5) uh, (0.5) m-me gusta la bicicleta mas, uh, >creo que< la bidcleta uh, es, 

7 la, uh, es la seleccion, uh, mejor para una dudad como esa? Uh. Y, um, (2 sec.) Y uh, necesitamos 

8 pensar en que, uh, en que mas razones para eso. Um, ((laugh)) uh, la polucion y la arreglacion, 

9 um, y uh. Tambien la biddeta da mas ejerdcio, um, a la persona que monta? Que lo monta? La 
10 monta. Um, y la motocicleta no da, uh, ejerdcio porque en la motodcleta, uh, solamente uh, so- 
il lamente: una persona se sienta, en eso. 
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'So, uh, my friends, I think that, uh, the bicycle is, is the, uh, better, solution, 
for the problem of traveling. Because, uh, I think that the bicycle? Uh, doesn't 
give, uh, more pollu — pollution, to the city, in Mexico, and um the, the mo- 
torcycle, does, um, give, the, pollution, um, the. Although the bi — the motor- 
cycle is, is faster, um, for traveling, the bicycle, doesn't require, uh, doesn't re- 
quire time or money, or too much time or money to fix and to be sure that, 
uh, it is working, uh, all the time. Um, uh, I — I like the bicycle more, uh, I 
think that, the bicycle uh, is, the, uh, is the best, uh, option for a city like this 
one? Uh. And um. And uh, we need to think about, uh, about more reasons 
for this. Um, (laughter), uh, the pollution and the fixing, um, and uh. Also 
the bicycle gives more exercise, um, to the person who rides? Who rides it? 
Rides it. Um, and the motorcycle doesn't give, uh, exercise because on the 
motorcycle, uh, just, uh, a person just sits, on it.' 



Mary's sample, formulated 
more in terms of stating and sup- 
porting an opinion than listing ad- 
vantages and disadvantages, shows 
much repetition, very little elabora- 
tion, no disadvantages for bicycles, 
and no concluding statement. The 
lack of a concluding statement, how- 
ever, was not uncommon in the re- 
sponses. Because the testees had only 
2 minutes to speak and were cut off 
when the time limit was reached, 
and because most were not watching 
a clock while speaking, but instead 
were concentrating on their re- 
sponses, few actually produced a 
complete "paragraph" in terms of 
having a beginning and an end. 



If one looks at the way in which 
Mary changed the task to that of giv- 
ing and supporting an opinion, 
however, it is evident that she was 
aware that she needed to support her 
opinion with reasons (Lines 7-11). In 
other words, she demonstrated her 
knowledge of an organizational 
structure, albeit for a different task, 
but had trouble filling it in. Thus, 
Mary's paradigm is not that of our 
proposed model for comparing ad- 
vantages and disadvantages. She 
performed well in the organiza- 
tional structure of her points, but ac- 
cording to a different model with a 
different function, that of stating and 
supporting an opinion, as shown in 
Table 3. 



Table 3 

Mary's Propositional Organization for Comparing 
Advantages and Disadvantages 

Topic sentence: La bicicleta es la solucion mejor para el problema de viajar 
'The bicycle is the best solution for the problem of traveling' 



(Continued) 
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Table 3 (Continued) 

Advantage 1 of bicycle: La bicicleta no da mas polucion 'The bicycle doesn't 
give more pollution' 

Disadvantage 1 of motorcycle: La motocicleta si da la polucion 'The motor- 
cycle does give pollution' 

Advantage 1 of motorcycle: La motocicleta es mas rdpido 'The motorcycle 
is faster' 

Advantage 2 of bicycle: La bicicleta no requiere tiempo o dinero 'The bicy- 
cle doesn't require time or money' 

Conclusion (?): Me gusta la bicicleta mas T like the bicycle more' 

—Necesitamos mas razones 'We need more reasons' 

Advantage 3 of bicycle: La bicicleta da mas ejercicio 'The bicycle gives more 
exercise' 

Disadvantage 2 of motorcycle: La motocicleta no da ejercicio 'The motorcy- 
cle doesn't give exercise ' (Una persona solamente se sienta) ('A person 
just sits') 

Conclusion: — 



The response from Deb (Ex- 
ample 3), who is a much less profi- 
cient speaker than the other two, il- 
lustrates a much more fragmented 
and less cohesive structure, typical of 
the "skeletal group," as seen in Table 
4. 



Deb's reply for comparing ad- 
vantages and disadvantages begins 
with two general statements that 
were probably intended to contrast 
with each other, but do not do so 
(Lines 1-3). They seemingly organize 



Example 3. Deb's Reply for Comparing Advantages and Disadvantages 

1 Deb: Bueno (1.0). Hay dos, dos cosas. Uno es que, que la gente debe montar, a una: bicicleta, para: 

2 para: trans — la transportation^ uh, y: otro es, los motocicletas son mejores, o mejor manera, 

3 una me — jor, una mejora manera, de, de, de (0.5) transpor — te, de transportation. Y, y creo que, 

o 
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4 que, los, las bicicletas son mas sanas para el ambiente. Y, y: no hacen, humo, ni nada en en el 

5 aire. Y, eso es una, buen caracterfstica porque: estamos en Mej — en la ciudad de Mexico, donde: el 

6 humo y todo de: de la industria es, (0.5) uh, malisima. Y: los motocicletas, uh, tienen, tienen, 

7 ventajas? Y desventajas tambien. Las ventajas son que: que es una: (0.2) un, un tipo, una tipa de 

8 transportacion mas rapida, y toda la gente puede: puede uh, (0.5) movar o, o, (0.5) si, mas 

9 rapida. Mas rapido. Y, la, la otra cosa es que: la gente no suda, cuando, cuando monta una moto- 

10 cicleta. En, en una bicideta si, todos Sudan. Y esa es una, una, buen, buena aventaja, para todos 

11 los que trabajen. 

'Well. There are two, two things. One is that, that people should ride, a bicy- 
cle, for, for, trans — transportation, uh, and, the other is, motorcycles are bet- 
ter, or a better means, a better, a better means, of, of, of, transport, of transpor- 
tation. And, and I think that, that, the, the bicycles are healthier for the envi- 
ronment. And, and, they don't make, smoke, or anything in the air. And, that 
is a, good characteristic because, we are in Mex-, in Mexico City, where, the 
smoke and all of, of the industry is, uh, really bad. And, the motorcycles, uh, 
have, have, advantages? And disadvantages too. The advantages are that, that 
it is a, a type, a type of faster transportation, and all the people can, can uh, 
move or, or, yes, more quickly. More quickly. And the, the other thing is that, 
people don't sweat, when, when they ride a motorcycle. On, on a bicycle yes, 
everyone sweats. And that is a, a good, good advantage, for all of those who 
work.' 



Table 4 

Deb's Propositional Organization for Comparing 
Advantages and Disadvantages 

Topic sentence: Hay dos cosas: (a) la gente debe montar una bicicleta para 
transportacion; (b) los motocicletas son mejor manera de transportacion 
'There are two things: (a) people should ride a bicycle for transportation; 
(b) motorcycles are a better means of transportation' 

Advantage 1 of bicycles: Son mas sanas para el ambiente "They are health- 
ier for the environment' 

(No hacen humo) ('They don't make smoke') 

(Es una buen caracteristica porque estamos en la ciudad de Mexico) ('It 
is a good characteristic because we are in Mexico City') 

(Continued) 
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Table 4 (Continued) 

Topic sentence 2: Los motocicletas tienen ventajas y desventajas tambien. 
'Motorcycles have advantages and disadvantages also' 

Advantage 1 of motorcycles: Una tipa de transportacion mas rdpida. 'A 
kind of faster transportation' 

(La gente puede movar mas rdpida) ('People can move more quickly') 
(La gente no suda ) ('People don't sweat') 

Disadvantage 1 of bicycles: Todos sudan 'Everyone sweats' 

[Referring to Advantage 1 of motorcyles above] Esa una buena aventaja 
para todos los que trabajen. 'It is a good advantage for all those who 
work' 

Conclusion: — 



the answer by stating the two points 
she wants to elaborate. She does give 
details on why people should ride 
bicycles (Lines 3-6), but instead of 
elaborating on why motorcycles are a 
better means of transportation, she 
goes on to say that they have advan- 
tages and disadvantages (Lines 6-7). 
She only states the advantages, how- 
ever. She shows a notion of struc- 
ture in her reply, but the content of 
her answer, what she actually says in 
terms of propositions, interferes 
with the unity of the sample, mak- 
ing it difficult to follow. In addition 
to her structural problems, she re- 
veals a definite lack of key vocabu- 
lary and shows problems with 
agreement, which she self-corrects. 
All of these problems contribute to 
her difficulty in expressing many 
propositions and supporting state- 
ments (see Table 4). 

Thus, the three examples reflect 
distinct levels of structural quality of 
answers in our data: complete, fairly 



complete, and skeletal. The groups 
are especially distinguishable when 
matched against an ideal model that 
represents a complete and coherent 
reply and a top-down organization, 
as seen in Tables 1A and IB. 

After analyzing each response 
for the various elements of a topic 
sentence, propositions of advantages 
and disadvantages, and supporting 
elaboration, as well as checking for a 
change of the task from that of com- 
paring advantages and disadvan- 
tages to that of giving and support- 
ing an opinion, we grouped samples 
according to patterns that emerged. 
What we found was a continuum of 
structural differences that could be 
characterized as shown in Table 5. 
These descriptions, still in various 
stages of development, are based on 
the models 1A and IB we proposed 
earlier, and show where learners di- 
verge from them. We should note 
that these categories should not be 
seen as tied to the ACTFL Guide- 
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lines. We have grouped them into 
three classes, labeled High, Mid, and 
Low, with three subcategories within 
each. On one end of the continuum 
we see a very complete response, in- 
cluding topic sentence, advantages 
and disadvantages of both referents, 
and a conclusion, all presented in a 
logical manner and approximating 
one of our two models. Then, at the 
High- level we see a topic sentence 
but fewer propositions than in the 
High+ and High-Mid levels. The tes- 
tee may also have changed the topic 
sentence from a statement that there 
are advantages and disadvantages to 
a statement of an opinion about 
which one is better (for example. 
Hay muchas ventajas y desventajas, 
pero yo creo que la bicicleta es mejor 
por varias razones 'There are many 
advantages and disadvantages, but I 
think that the bicycle is better for 



various reasons'). The response, 
however, includes advantages and 
disadvantages of both referents. 

In the next two levels the tes- 
tees commonly change the task from 
that of comparing advantages and 
disadvantages to that of giving and 
supporting an opinion. This change 
in task is understandable if one con- 
siders that, by stating a preference for 
one referent, such as bicycles, the tes- 
tee can avoid having to account for 
the advantages and disadvantages of 
both referents, thus simplifying the 
task. In the Mid+ level, the task is 
changed and the testees state propo- 
sitions actually supporting only one 
side of an opinion. The Mid-Mid tes- 
tees give fewer advantages and dis- 
advantages, or only present either 
advantages or disadvantages. There 
is a notable lack of development in 
the ideas, so that the testee seems to 



Table 5 

Characteristics of the Three Proficiency Levels and Their Sub- 
categories for Comparing Advantages and Disadvantages 

High (More Complete! 

High+ Has topic sentence, both advantages and disadvantages of each 
referent, and a conclusion, in a coherent and cohesive organiza- 
tion, closely following and completing an organizational model 
(as in 1A or IB). 



High-Mid Has topic sentence and several advantages and disadvantages, in 
a coherent presentation of ideas, following a partially complete 
organizational model. 

High- Has topic sentence and some advantages and disadvantages are 

given. One or two propositions may not fit tightly in a coherent 

(Continued) 
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Mid+ 


Table 5 (Continued) 

presentation of ideas. Possible change of task from comparison of 
advantages and disadvantages to stating and supporting an opin- 
ion. 

Mid (Fairlv Complete) 

Has topic sentence, but changes task to that of stating and sup- 
porting an opinion. Several propositions are given that really 
support only one side of the argument. 


Mid-Mid 


Has topic sentence, but changes task to that of supporting an 
opinion. Response not complete, either with too few proposi- 
tions expressing advantages and disadvantages or only a present- 
ing a few advantages or disadvantages. Lacks development of re- 
sponse either because of too much elaboration on only one or 
two points or extraneous material inserted that does not directly 
support the reply. 


Mid- 


Has topic sentence but changes task to an opinion, with some 
advantages and disadvantages that are not logical or consistent 
in their support of the opinion. 

Low (More Skeletal) 


Low+ 


Has topic sentence, but there are some incoherencies in the sam- 
ple, either in the statements of the advantages and disadvantages 
or in the development of the response. 


Low-Mid 


Has topic sentence, but answer is incomplete, with few advan- 
tages and disadvantages. Some statements are incoherent and 
difficult to follow in logic. Language difficulties in vocabulary 
and grammar notably affect the ability to answer. 


Low- 


May or may not have topic sentence. Answer is incomplete due 
to difficulties in vocabulary and grammar, with perhaps only 
one advantage and/or disadvantage. 



be simply listing. The Mid- level the beginning of some problems of 

gives some advantages and/or dis- logic and consistency, 

advantages that do not support the Problems of logic and consis- 

topic sentence well, so one sees here tency are seen notably in the Low 

level of testees. Difficulties with the 
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vocabulary and grammar definitely 
affect the testees' ability to answer 
the stimuli at all. Some statements 
are incoherent, and the logic of the 
propositions is difficult to follow. At 
the low end of the continuum, the 
response may or may not have a 
topic sentence, and perhaps only one 
or two propositions are stated in the 
2 minutes allotted. 

Of the 24 included in this study, 
the testees were divided as shown in 
Table 6. Most of the testees placed in 
the Mid level, and relatively few in 
the Low level. 

IMPLICATIONS FOR TEACHING 

Grading oral production is a 
problem for all levels of foreign lan- 
guage teachers, especially at the 



more advanced levels, where we of- 
ten encounter learners who have 
different backgrounds and prior ex- 
periences with the target language. 
In a conversation course, the teacher 
faces the problem of grading the 
learners' achievement for that 
course and not their level of profi- 
ciency, although the ultimate goal, 
which is not always attainable in a 
single semester, is to see some gain 
in proficiency. If one grades oral pro- 
ficiency alone, then those students 
who have a relatively high level of 
proficiency prior to taking the 
course — from having traveled 
abroad or by heritage, for example — 
will almost always demonstrate a 
higher level of proficiency than 
those without such a background. 



Table 6 

Divisions of the 24 Testees According to the Grading Criteria 



High 


Mid 


Low 


Level 


No. of 
Testees 


Level 


No. of 
Testees 


Level 


No. of 
Testees 


High+ 


2 


Mid+ 


3 


Low+ 


2 


High-Mid 


4 


Mid-Mid 


6 


Low-Mid 


1 


High- 


2 


Mid- 


3 


Low- 


1 



and they may not have to study for 
the course at all. On the other hand, 
if one grades only on the attainment 
of particular knowledge of the 
course content, then the teacher 
cannot grade the test results in a ho- 



listic manner. Often, this focus on 
particular knowledge means that the 
teacher downplays the importance of 
what the learners are actually saying 
in their responses. 




1 
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What we propose is that the 
advanced conversation teacher grade 
for both proficiency and achieve- 
ment. In some respects, we are pro- 
posing a new kind of prochievement 
test, but not in the sense that the 
learner must show knowledge of 
discrete items. Instead, the learner 
must demonstrate proficiency in 
performing given functions, such as 
comparing advantages and disad- 
vantages, in a cohesive, coherent, 
and organized manner. It follows, 
then, that, since course tests must 
reflect what is taught, a large part of 
the learner's achievement should be 
mastery of organizational structure. 
This focus then implies that an im- 
portant component of the advanced 
conversation course be teaching 
good organizational templates for 
the tasks targeted at the Advanced 
and higher levels, such as those seen 
in the models of Tables 1A and IB. If 
learners can be given a set of dis- 
course templates that they can rely 
on to fill in propositions and if 
teachers listen for their skill in ap- 
plying those templates, then we be- 
lieve the learners will have better 
guidance in producing top-down 
and, hopefully, fairly complete an- 
swers. 

These templates can be easily 
learned. In our course there was a 
de-emphasis of vocabulary specific to 
a given topic, since the TOPT con- 
fronts the learner with any topic re- 
lated in any general way to what 
Spanish teachers in the U.S. would 
need to know or say in their teach- 
ing. In a typical (non-TOPT) ad- 
vanced conversation course, how- 
ever, the teacher could also teach 
and test for specific vocabulary, such 
as connective adverbials. We would 



hope, nevertheless, that discourse 
structure be considered one of the 
most, if not the most, important 
element to be mastered in the 
course. 

One may argue that these tem- 
plates are the same as those used in 
English and, therefore, we are not 
teaching the learners anything new. 
Experience, as well as our data, how- 
ever, has shown that learners do not 
follow these frameworks under 
pressure in the target language. 
Learning how to express a topic sen- 
tence in a foreign language is a chal- 
lenge, especially one that is clear and 
to the point of the question and that 
presents a good proposition for de- 
velopment. 

Finally, we are proposing to 
move the basis of testing for the 
conversation class to the level of the 
discourse itself, thereby contextualiz- 
ing the vocabulary and discrete 
grammar points. Learners can pro- 
duce more coherent speech, and 
those who already have a fairly good 
oral proficiency upon entering the 
course can sound even more pol- 
ished and articulate. Our recom- 
mendation for a top-down struc- 
tured approach to discourse assess- 
ment is based on our encourage- 
ment of a top-down approach to the 
four skills of listening comprehen- 
sion, reading, writing, and, as we 
have discussed here, speaking. 

NOTES 

1 One could argue that the logical 
conclusion to a response compar- 
ing advantages and disadvan- 
tages is an opinion and, in fact, 
that is how the model answers 
conclude in 1A and IB. The ques- 
tion in the stimulus itself ("What 

18 
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do you think?") may also seem to 
elicit an opinion, despite the em- 
phasis on comparing advantages 
and disadvantages. In such a per- 
spective, the organization would 
seem to be bottom-up, rather 
than top-down, with the opinion 
stated at the end. We point out, 
however, that an opinion is not 
the objective of the task, and that 
this task is answered most suc- 
cessfully with a topic sentence at 
the very beginning that guides 
the construction of the rest of the 
response. Moreover, testees 
rarely have time to give a conclu- 
sion at the end of their response, 
a fact that would also argue 
against a bottom-up analysis. 

2 The notations used in the tran- 
scriptions are those found in 
Atkinson and Heritage (1984). 

3 These translations do not show 
errors such as vocabulary or 
agreement errors. 
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